Method of modifying an image on a computational device

ABSTRACT

A method of modifying an image on a computational device and a system for implementing the method is disclosed. The method comprising the steps of: providing image data representative of at least a portion of a three-dimensional scene, the scene being visible to a human observer from a viewing point when fixating on a visual fixation point within the scene; displaying an image by rendering the image data on a display device; capturing user input by user input capturing means; modifying the image by: computationally isolating a fixation region within the image, the fixation region being defined by a subset of image data representing an image object within the image, wherein the image object is associated with the visual fixation point; spatially reconstructing the subset of image data to computationally expand the fixation region; spatially reconstructing remaining image data relative to the subset of image data to computationally compress a peripheral region of the image relative to the fixation region in a progressive fashion as a function of a distance from the fixation region, wherein the modification of the image is modulated by the user input such that a modified image is produced which synthetically emulates how the human observer would perceive the three-dimensional scene.

The present invention relates to a method of modifying an image on acomputational device. In particular, the invention relates to a methodfor modifying still and moving images recorded from cameras oncomputational devices.

Computational devices equipped with cameras, sensors and touchscreens,such as photographic cameras, movie cameras, smartphones, and tablets,are increasingly being used to record and manipulate photographs andmovies. Conventional optical and image processing methods used in suchdevices generally rely on the geometry of linear perspective to map thelight rays from the three dimensional (3D) world entering the lens orlenses mounted on the devices to a two-dimensional (2D) image plane. Thecomputational device can modify various properties of this linearperspective 2D image by using standard optical, geometrical and imageprocessing methods. For example, the 2D image can be zoomed, cropped,warped, stretched, rotated or filtered in order to satisfy therequirements of the user by applying user-controlled algorithms or otherprocesses known by those skilled in the art.

However, there are a number of problems and limitations associated withcamera devices that rely on linear perspective to capture and displayimages of the 3D world. Linear perspective operates on the principlethat light rays travel in straight lines through a pinhole in a lightbarrier to be projected onto an image plane. For narrow angles of view(<30° horizontally) the appearance of any object as projected onto theimage plane and rendered to a display according to the geometry oflinear perspective is relatively unaffected by geometric distortion.However, a limitation with images of narrow angles of view is that largeareas of the scene being imaged are cropped, and so not recorded in theimage. As the angle of view is increased (>60° horizontally) then moreof the scene can be recorded in the image, but due to the geometry oflinear perspective objects at the edges of the image will start tobecome increasingly stretched and objects at the centre will start to beminified in a way that appears for many people perceptually unnatural.As the angle of view is increased further (>100° horizontally) the abovenoted distortions will become more severe, to the point where images ofangles of view of >160° horizontally will be increasingly illegible.

Other methods can be used to obtain wide-angle views of 3D scenes thatavoid the distortions associated with standard linear perspectiveprojections. These include the use of fisheye lenses, 360° lenses, andpanoramas. However, all these methods introduce different kinds ofdistortion, which is not desired and disturbs accuracy of the mappingbetween the real 3D scene and the computer image. Distortion problemswith such methods can be reduced if the image is mathematicallyprojected onto a spherical or curved surface and viewed using anappropriate interface, such that the user can scan around the imagewhile seeing a cropped portion of the image in a more naturalisticperspective, as is the case in 360° videos and similar technologies. Butthese methods revive the problem of restricting the view of a scene to anarrow angle within the frame of the viewing device.

In theory, wide-angled images generated using the principles of linearperspective could appear undistorted if the image were to be viewed froma sufficiently close distance, that is, at the correct centre ofprojection of the light rays. The reason is that at this point thepattern of light rays reaching the retina of the viewer would closelymatch the pattern of light paths than would be projected from the realscene. However, in most practical situations, and especially forwide-angle images, the correct centre of projection is too close to theimage surface for the normal human eye to focus comfortably. Therefore,linear perspective projections are not a viable method for accurately orcomfortably depicting objects in wide angled views.

In whatever way they are viewed, non-stereoscopic images generatedaccording to the principles of linear perspective characteristicallyappear flat when compared to our experience of the 3D real world theyrepresent. This is for several reasons, including the fact that thedepth cues in the image (such as occlusion, parallax, shading, etc.) aresuppressed by the cues showing the image is flat (such as the geometryof the image plane, surface glare from the screen, etc.). A furtherreason is that the organisation of the visual space in images generatedby conventional geometric methods is not perceptually natural, as notedabove, so limiting the amount of depth perceived in the image.

Computational devices equipped with cameras and touchscreens, or camerasequipped with computational hardware and touchscreens, are increasinglybeing used to make images that emulate the first-person point of view.However, the spatial structure of the 3D world experienced by a personwith normal binocular vision of their own point of view is substantiallydifferent in structure from images generated by conventional methods,that is, linear perspective or fisheye lens projection. It is known thatthe perceptual structure of visual space cannot be represented on animage plane using the geometrical laws of linear perspective, whichlimits any device reliant on those laws from effectively representingthat structure.

Computational devices equipped with cameras are increasingly beingfurther equipped with systems that permit the capture of 3D depthinformation from the scene, whether this is through the provision ofadditional cameras, light field capture systems, laser or infra-redmeasuring systems, time of flight systems, depth from motion systems, orother systems. The addition of 3D depth data can, in principle, lead toimprovements in the perceptual naturalness, perceived depth of images asit allows greater ability to simulate the appearance of objects innatural vision and to computationally manipulate the spatial propertiesof the image. However, current systems using depth capture technologytend to rely on the geometry of linear perspective to project theresulting 3D images to the 2D display on the device, thus reviving thelimitations noted above.

Furthermore, conventional user interfaces on smartphones and tablets, orother computational image capture devices equipped with touchscreens, donot allow the user to manipulate the spatial, depth and properties ofimages in a way that overcomes the problems or limitations noted above,in order to generate images that are more perceptually natural thanthose created or projected according to conventional projectivegeometry.

There is a need for a method that allows a user of a computationaldevice equipped with a camera and a touchscreen to take a photograph ormoving image of a 3D scene, whether using 2D or 3D depth data, that canbe manipulated in such a way as to improve the perceptual naturalness,perceived depth and effectiveness of the first-person viewpoint comparedto current methods.

U.S. patent application Ser. No. 14/763,454 discloses a method of makingan image of a scene (including a scene made by the method) generallycorresponding to that perceived by the human brain via the human eyes,the method including the steps, in any suitable order, of: capturing,recording, generating, or otherwise representing a scene consisting ofthe entire field of view, or part thereof, visible to a human observerfrom a given ‘Viewing Point’ (VP) when fixating on a given region withinthe scene, progressively enlarging the image towards the area of thescene, and progressively compressing the area of the scene correspondingto the peripheral field of vision to thereby produce a modified image ofthe scene generally corresponding to how the scene would appear to thehuman perceiver.

It is an object of the present invention to provide a technical solutionto at least some of the issues outlined above.

In accordance with a first aspect of the present invention, there isprovided a method of modifying an image on a computational device, themethod comprising the steps of: providing image data representative ofat least a portion of a three-dimensional scene, the scene being visibleto a human observer from a viewing point when fixating on a visualfixation point within the scene; displaying an image by rendering theimage data on a display device; capturing user input by user inputcapturing means; modifying the image by:

computationally isolating a fixation region within the image, thefixation region being defined by a subset of image data representing animage object within the image, wherein the image object is associatedwith the visual fixation point;

spatially reconstructing the subset of image data to computationallyexpand the fixation region;

spatially reconstructing remaining image data relative to the subset ofimage data to computationally compress a peripheral region of the imagerelative to the fixation region in a progressive fashion as a functionof a distance from the fixation region, wherein the modification of theimage is modulated by the user input such that a modified image isproduced which synthetically emulates how the human observer wouldperceive the three-dimensional scene.

In an embodiment, modifying the image may further comprise rotating theimage around an axis of the fixation region.

In an embodiment, modifying the image may further comprise applying ageometrical transformation to the image.

In an embodiment, modifying the image may further comprise altering anangle of view of the image in a horizontal or vertical axis of theimage.

In an embodiment, the fixation region may be computationally expanded byan amount positively correlated with the angle of view of the user.

In an embodiment, the peripheral region may be computationallycompressed by an amount inversely correlated with the angle of view ofthe user.

In an embodiment, the user input may comprise motion data representing amotion of the user relative to the display device. Modifying of theimage may comprise statically positioning the fixation region relativeto a border of the image and moving the peripheral region relative tothe fixation region in accordance with the motion of the user so as toemulate a motion parallax perceived by the human observer in thethree-dimensional scene. Modifying of the image may comprisecomputationally isolating the fixation region in response to anindication gesture of the user. Capturing of the user input may comprisemonitoring a movement of the eyes of the user which corresponds to arepositioning of the fixation point within the three dimensional sceneand modifying the image comprises computationally isolating the fixationregion comprising a repositioned fixation point.

Capturing of the user input may comprise monitoring touch gestures on atouchscreen of the display device which correspond to a repositioning ofthe fixation point within the three dimensional scene and modifying theimage comprises isolating the fixation region comprising a repositionedfixation point.

In an embodiment, the image data may comprise data relating to adistance between objects and the viewing point in the three dimensionalscene.

In an embodiment, computationally isolating the fixation region maycomprise computationally processing the subset of the image data so asto determine boundaries of at least one image object associated with thevisual fixation point, isolating the at least one image object andrendering each image object on a separate depth layer.

In an embodiment, the method may further comprise a step of updating thedisplay of the display device subsequent to each step of modification ofthe image.

In an embodiment, the image data may be generated by an optical devicecomprising a lens selected from a group comprising rectilinear cameralens, fisheye camera lens, 360° lens, multiple lenses and mechanicallyadjustable lens.

In an embodiment, the method may further comprise a step ofcomputationally processing the image data so as to apply an image blureffect which progressively increases radially away from the fixationpoint in at least one of height, width and depth axes of the image.

In an embodiment, the method may further comprise a step ofcomputationally processing the image data so as to overlay the imagewith an object seen in proximity to the face of the human observer inthe three dimensional scene.

In an embodiment, capturing the user input may comprise computing adistance between the image and a head of the user. The fixation regionmay be computationally expanded by an amount inversely correlated withthe distance between the image and the head of the user. The peripheralregion may be computationally compressed by an amount positivelycorrelated with the distance between the image and the head of the user.

In accordance with a second aspect of the present invention, there isprovided a computer system configured to implement steps of the methodaccording to the first aspect, the system comprising: user inputcapturing means configured to capture user input; a control unitconfigured to generate a processed image data based on the captured userinput; a display device configured to display the processed image data.

In an embodiment, the system may further comprise image capturing meansconfigured to capture the image data which represents a threedimensional scene.

In an embodiment, the system may further comprise a depth sensorconfigured to capture depth information from the three-dimensional sceneand wherein the control unit is configured to process the captured imagedata along with the captured depth information.

In an embodiment, the user input capturing means may comprise a displaydevice motion sensor configured to capture motion applied to the displaydevice.

In an embodiment, the user input capturing means may comprise a usermotion sensor configured to capture motion of the user relative to thedisplay device.

In an embodiment, the user input capturing means may comprise atouchscreen configured to be integrated with the display device.

In an embodiment, the system may further comprise a graphics processorconfigured to process the captured image data so as to generate amodified image data.

Whilst the invention has been described above, it extends to anyinventive combination of features set out above or in the followingdescription. Although illustrative embodiments of the invention aredescribed in detail herein with reference to the accompanying drawings,it is to be understood that the invention is not limited to theseprecise embodiments.

Furthermore, it is contemplated that a particular feature describedeither individually or as part of an embodiment can be combined withother individually described features, or parts of other embodiments,even if the other features and embodiments make no mention of theparticular feature. Thus, the invention extends to such specificcombinations not already described.

The invention may be performed in various ways, and, by way of exampleonly, embodiments thereof will now be described with reference to theaccompanying drawings, in which:

FIG. 1 illustrates a flow chart of the steps of the method of modifyingan image on a computational device according to a first embodiment ofthe present invention;

FIG. 2 illustrates a block diagram of a computer system implementingsteps of the method of FIG. 1;

FIG. 3 illustrates an example explaining geometric principles of linearperspective;

FIG. 4 illustrates the image modification according to the steps of themethod of FIG. 1;

FIG. 5 illustrates a schematic representation of Euclidean space;

FIG. 6 illustrates a spatial modification according to the presentinvention;

FIG. 7 illustrates a schematic representation a 3D scene generated by alinear perspective projection in a camera or similar device;

FIG. 8 illustrates an example scene showing a step of the method of FIG.1;

FIG. 9 illustrates another example scene showing a step of the method ofFIG. 1;

FIG. 10 illustrates an example scene showing a step of the method ofFIG. 1;

FIG. 11 illustrates an example scene showing a step of the method ofFIG. 1;

FIG. 12 illustrates an example of a User Interface to be used in anembodiment of the present invention.

A method 100 of modifying an image on a computational device accordingto a first embodiment of the present invention will now be describedwith reference to FIG. 1. At step 101, the image data from the camera,camera array, and depth sensors where available, is accessed. At step102, a fixation region 102 a is defined as a combination of fixationcoordinate, this corresponding to the point in the scene where the eyeor eyes are fixating, and the fixation object, this being an object inthe scene associated with the fixation coordinate. The fixation regionmay be selected by one of three techniques 102 b: a default setting, byuser gesture, or via the eye tracking sensor integrated into the device.

The default setting may be applied using a suitable algorithm,programmed into the device by a person skilled in the art, which selectsa region in the centre of the image, or close to the centre of theimage, or detects a salient feature in the image, such as a person or aface of the person, or an object in the centre of the image, and selectsthis as the fixation region.

The user gesture selection may be enabled by the user applying asuitable gesture to the touchscreen display, on which the image isoptionally displayed at step 103. For example, the finger of the usertouches an area of the touchscreen and thereby selects that point in theimage as the fixation region. Moving the finger across the touchscreenmoves the fixation region correspondingly.

The eye tracking selection is enabled once the image is displayed to thescreen and a suitable sensor integrated into the device embodying themethod detects the motion and position of the eye of the user and withrespect to the display and uses the position of the gaze on the image toselect the fixation region. Moving the gaze across the image moves thefixation region correspondingly.

At step 104, the image is modified according to the following way. Theregion of the image corresponding to the fixation region, this havingbeen selected in a prior step, is magnified 104 a relative to the sizethereof in the original projection of the image generated by the linearperspective geometry applied to the image data from the 3D scene. Theremainder of the image is minified 104 b relative to its size in theoriginal projection of the image generated by the linear perspectivegeometry applied to the image data from the 3D scene. There are severalcomputational techniques that can be used to effect the spatialmodifications specified, such as expansion (magnification) orcompression (minification) in the present invention, and which could beimplemented by a person skilled in the art. By way of example only, onetechnique is to apply a suitable mesh transformation to the 3Dcoordinates used to model the 3D scene, or the 2D image of the 3D scene.Another technique is to apply a suitable matrix transformation to thecomputer data used to represent the light paths in the 3D scene.

At step 105, the image modified according to the first embodiment isrendered to the image display, which may also be a touchscreen device.The touchscreen may provide a user interface 105 a, in the form ofbuttons, sliders, switches, or similar user interface mechanisms, and bymanipulating this interface the user is able to effect furthermodification to the image. According to the disclosed method theseinclude further selecting or modifying the fixation region, isolating aobject in the scene corresponding to the fixation region, altering theangle of view of the image, whether in the horizontal or vertical axis,rotating the image around the axis of the fixation region, or anotheraxis as specified by the user via a suitable interface controlmechanism, altering the curvature in the image, warping the image, orapplying other geometrical transformations to the image. Once selectedthese modifications update the image on the image display in real time.

At step 106, further modifications to the image can be made in responseto data passed from the sensors 106 a integrated into the device, suchas the eye tracker, head tracker, accelerometer, or gyroscope, such thatthe image is modified in real time in response to user behaviour,gestures, motion, or the motion of the device.

At step 107, image effects 107 a may be applied to modify the image.These effects may be applied using an algorithm programmed into thedevice by a person skilled in the art, such they modify the imageautomatically without user intervention, or may be applied under thecontrol of the user via a suitable interface, which may be controlledvia the touchscreen. Image effects may include blur, modifications toimage contrast, saturation, luminance, and resolution. Additional imageeffects 107 b may be overlaid on the image automatically or under thecontrol of the user that are designed to enhance the perceptualauthenticity of the first person point of view. Items included wouldnormally be seen in natural vision in close proximity to the eyes, and anon-exhaustive list of examples would include spectacles, sunglasses,hats, noses, jewelry, hair, body piercings, and umbrellas.

The method of modifying an image on a computational device according toa first embodiment may be embodied in a computer system 200 asillustrated in FIG. 1. Cameras or camera arrays 201 may record rays oflight from a 3D scene by projecting them onto a sensor or sensors (notshown) using the geometry of linear perspective and converting them tocomputational data 201 a representing luminance, colour, motion, etc.which may then be stored. In addition, depth information 202 a from thescene can be recorded by using sensors 202, by calculating the disparitybetween multiple cameras, from camera motion, by applying light fieldcapture techniques, or other techniques known to those skilled in theart, this information being converted to computational data and storedin the device. Camera lenses may be rectilinear or fisheye inconstruction, and the device may record a cropped region of the 3D sceneor a 360° view of the 3D scene, in mono or in stereo, in a flatprojection or using 3D depth information about the 3D scene, or using alight field system, or similar system for recording light arrays in a 3Dscene.

The data representing the 3D scene is accessed by the Central Processor203 and modified according to a number of steps so as to obtain amodified data 203 a, discussed above with reference to FIG. 1. Themodified image data 203 a is passed to the Graphics Processor 204 forfurther modification 204 a according to the steps disclosed withreference to FIG. 1, and rendered to the image display 205 which mayalso be a touchscreen display, so as to obtain a rendered image 205 a.

Data 205 b from the touchscreen display 205 generated in response touser behaviour (such as finger gestures) can be passed to the CentralProcessor 203 in order to initiate further modifications to the imageaccording to steps specified with reference to FIG. 1.

The image may be further modified in response to data derived frommotion sensors 206, such as gyroscopes, or accelerometers 206 aintegrated into the device, or from sensors 207 that detect the motion,position or behaviour of users, such as eye tracking or head trackingsensors 207 a. Once modified it is passed again to the GraphicsProcessor 204, via the Central Processor 203, and rendered to thedisplay 205.

The system 200 is so configured as to permit the user to continuouslymonitor the modified image via the display 205 and further modify itsproperties in real time, enabled by user gesture inputs or in responseto motion, eye, or head tracking sensors 207, 207 a.

FIG. 3 illustrates the geometric principles of linear perspective. Theimage on the screen at FF shows a wide-angle view (120° horizontally) ofthe room, which is shown in plan view at AA. Note that due to the wideangle of view of the 3D scene the cube in the room at BB appearsexcessively small in the image at BB′, while the walls of the roomappear excessively stretched at AA′. The forward-most cube at CC appearsexcessively large when projected to CC′, and when the standard cameracrop frame, indicated by the dashed line at GG, is applied much of theroom falls outside the visible area. For the purposes of representingthe room in a perceptually natural way, this method is inadequate.

In theory, the apparent distortions of size and shape in this image ofthe room could be neutralised if the viewer adopts the correct centre ofthe projection as the viewing point. Under these conditions the patternof light rays entering the pupil of the eye would closely approximatethose projecting from the real scene. However, for wide-angle views thecentre of projection would be too close to the image plane to allow theeye of the view to focus comfortably, unless the image was madeimpractically large.

FIG. 4 illustrates the image modification principle employed in thepresent invention. The same room as in FIG. 3 is shown but the paths ofthe rays of light projected onto the image plane at EEE are different,resulting in a modified image of the 3D scene. The rays of lightprojecting from the cube at BBB are non-linear is a way specified in thepresent invention, being more divergent at the aperture than in FIG. 3,resulting in a magnification of the cube at BBB′ relative to theprojection in FIG. 3. The rays of light projecting from the outer edgesof the room are more convergent at the aperture than in FIG. 3,resulting in a minification the outer walls of the room at AAA′. Sincethe cube CCC is closer to EEE than BBB, it is also minified whenprojected at CCC′ according to the present invention. The wall behindthe cube at BBB is also minified according to the present invention, andis shown in the drawing as smaller in the image at AAA′ in proportion tothe cube at BBB′ than in FIG. 3.

Note that in the projection of the room shown in FIG. 4 the entire roomis visible within the standard camera crop frame at GGG. This methodresults in an image of the scene that is more perceptually natural, morecomfortable to view, and has greater perceptual depth than an imagegenerated according to the geometry of linear perspective.

The degree of magnification and minification applied to the scene in thedrawing is shown for the purposes of illustration only, and does notindicate the precise or only degree of modification used in a deviceembodying the method.

The image of the 3D space shown in FIG. 4 does not suffer some of theproblems noted in images generated by the geometry of linearperspective, and is designed to improve the layout, legibility,perceptual depth, and perceptual naturalism of the image compared to theimage of the 3D scene generated in FIG. 3.

FIG. 5 is a schematic representation of Euclidean space. Threeintersecting planes representing a 3D space are shown in an orthographicprojection. Each plane is marked with evenly distributed grid lines toillustrate that principle that in Euclidean space 3D coordinates areevenly spaced throughout the volume. A standard linear perspectiveprojection would project these coordinates to a 2D plane using straightlight paths passing through an aperture and intersecting with a plane.

FIG. 6 is a schematic representation of the spatial modification appliedin the present invention. Three intersecting planes marked with gridlines representing a 3D space are shown in an orthographic projection,as in FIG. 5. Here the grid lines are not evenly distributed. Thecoordinate where all the planes intersect is taken to be the fixationpoint in the 3D space, and origin, and the space between the points isexpanded in this region, while the space beyond this region iscompressed in a way that increases as a function of distance from theorigin. When this 3D space is projected to a 2D plane it produces anon-Euclidean projection of the 3D space. Note that the cube at thecentre of the space is magnified in FIG. 6 when compared FIG. 5 whilethe space at the edges is increasingly minified.

The degree of magnification and minification applied to the scene in thedrawing is shown for the purposes of illustration only, and does notindicate the precise or only degree of modification used in the presentinvention. By way of example, a person skilled in the art would programan algorithm into the device that controls the degree of magnificationand minification either automatically or under user control.

There are several computational techniques that can be used to effectthe spatial modifications specified in the present method, and whichcould be implemented in a device embodying the present invention by aperson skilled in the art. By way of example only, one technique is toapply a suitable mesh transformation to the 3D coordinates used to modelthe 3D scene, or the 2D image of the 3D scene. Another technique is toapply a suitable matrix transformation to the computer data used torepresent the light paths in the 3D scene.

FIG. 7 shows a schematic representation 30 of a 3D scene generated by alinear perspective projection in a camera or similar device. Note thatthe tree 31 is identical in size to the tree 32 in the scene, butappears smaller in the image as it is further away from the projectionplane. A device embodying the present invention equipped with arectilinear lens, or equivalent optical apparatus, would generate animage of this kind.

FIG. 8 illustrates an example scene 300 with selecting and isolating 102a fixation region in the image of the 3D scene 300. The pattern-filledcircle 301 indicates the fixation coordinate located in the treelabelled 302, and the computational isolation of the tree from otherobjects in the scene is indicated by the dashed outline of the tree at302. The fixation coordinate 301 can be selected by a number oftechniques, including but not limited to: user control via thetouchscreen interface; automatically by applying a suitable algorithm;in response to sensors such as eye tracking systems. By way of example,the user touches the image on the touchscreen, and the point of contactwith the finger and the screen is taken as the fixation coordinate usingtechniques known to those skilled in the art.

The object associated with the fixation region can determined in anumber of ways including but not limited to: user control via thetouchscreen interface; automatically by applying a suitable algorithm;in response to sensors such as eye tracking systems. By way of example,the user draws a loop with a finger around an object in the image on thetouchscreen, and an object enclosed by the loop, whether 2D or 3D, istaken as the associated object using techniques known to those skilledin the art.

The object associated with the fixation region can computationallyisolated from other objects in the scene in a number of ways includingbut not limited to: user control via the touchscreen interface;automatically by applying a suitable algorithm; in response to sensorssuch as eye tracking systems. By way of example, the user draws aroundwith a finger around the contour of an object in the image on thetouchscreen, and an object enclosed by the contour, whether 2D or 3D, istaken as the isolated object, and computationally isolated usingtechniques known to those skilled in the art.

FIG. 9 shows an example scene 400. The step of magnifying 104 a thefixation region 301 and minifying 104 b the remaining regions andobjects in the scene 400 and the effects on the tree at 401 and 404, andthe flowers at 405 are shown. The dashed boundary at 406 indicates theoriginal angle of view of the image prior to modification according tothe first embodiment, with the regions inside the dashed line beingvisible in the pre-modified version illustrated in FIG. 8. The regionsoutside the dashed line now become visible within the frame 407. Thebold arrows 402 a, 402 b, 402 c, 402 d indicate the direction ofmagnification and the bold arrows 403 a, 403 b, 403 c, 403 d, 403 e, 403f indicate the direction of minification.

The tree at 401, now shaded, is larger in the image than in FIG. 8 andthe tree at 404 is smaller. The flowers 405 in the bottom right cornerof the image, which are closer that the tree at 401, are now smaller butalso visible in greater part than in FIG. 8. The dashed boundary at 406shows the portion of the 3D scene visible within the image frame in FIG.8, and FIG. 9 shows the additional areas of the 3D scene now visiblewithin the image frame.

The modifications applied to the image according to the presentinvention are designed to improve the perceptual naturalism and depth ofthe image of the 3D scene, and to improve its legibility.

The degree of magnification and minification applied to the scene in thedrawing is shown for the purposes of illustration only, and does notindicate the precise or only degree of modification used in a deviceembodying the method.

FIG. 10 illustrates a scene 500 showing moving the objects lying outsidethe fixation region. The shaded tree at 501 is the fixation region, andremains static with respect to the image frame 502, while the remainderof the objects in the scene are moved or otherwise modified. The drawingin FIG. 10 shows, by way of example, an effect of rotation 503 a, 503 b,503 c, 503 d about the axis of the fixation region. Other forms ofmotion or modification may be implemented, including but not limited to:rotation, translation, forward or backward motion, zooming, warping, orbending.

The type of motion used may be determined by the motion of the deviceembodying the method, and detected via a suitable sensor such as anaccelerometer or gyroscope such that the motion of the image correspondsto the motion of the device, thus enabling an effect of motion parallaxbetween the fixation region and the rest of the scene. This motionparallax effect can further enhance the perceptual depth in the image.

The type of motion used may be further determined by the eye or headmovement of the user of the device embodying the method, and detectedvia a suitable sensor such as an eye or head tracking system such thatthe motion of the image corresponds to the motion of the eyes or head,thus enabling an effect of motion parallax between the fixation regionand the rest of the scene.

FIG. 11 illustrates by way of example a scene 600 showing overlaying ofimage effects on the image in order to enhance the perceptual naturalismof the first person point of view represented by the image. By way ofexample only, the figure shows a pair 601 of spectacles overlaid on theimage of the 3D scene. The figure shows the spectacles 601 as they wouldappear to a wearer viewing them with two eyes, the image of which isfused by the visual system in a way approximated in the drawing. Theoverlaid images may be rendered with image blur and transparencyeffects, applied with techniques known to those skilled in the art, inorder to further emulate the perceptual appearance of objects seen inclose proximity to the eyes. In the case of the human nose, for example,this would appear in the image with a degree of blur and transparency inthe image, as it does in natural vision.

FIG. 12 illustrates an example of a User Interface 700. Such aninterface 700 may be equipped with a series of sliders or buttons,whether physical or virtual, that when altered send data to the CentralProcessor in order to effect modifications to the image or to controlthe behaviour of the device. In the example shown in this drawing, thereare seven sliders 701, 702, 703, 704, 705, 706, 707, which can returnvalues of between 0 and 100. By way of example, in one embodiment of theinvention, slider 701 controls the size of the fixation region, with 0being a small region and 100 being a large region; slider 702 controlsdegree of magnification applied to the fixation region, and slider 703controls the degree of minification applied to the non-fixation region;slider 704 controls the angle of view of the image, with 0 being anarrow angle (<20° horizontally) and 100 being a wide angle (>170°horizontally); slider 705 controls the degree of rotation of the image,with 0-50 being a leftward rotation and 51-100 being a rightwardrotation; slider 706 controls the apparent curvature in the image, 0being no curvature and 100 being that all straight lines in the 3D sceneare rendered as curves in the image, excepting lines of latitude in lineof sight or lines of longitude in line of sight; and slider 707 controlswhich first person image overlay effect is applied, 1 being spectacles,2 being sunglasses, 3 being noses, etc.

The invention claimed is:
 1. A method of modifying an image on acomputational device, the method comprising: providing, by a processor,image data representative of at least a portion of a three-dimensionalscene, the three-dimensional scene being visible to a human observerfrom a viewing point when fixating on a visual fixation point within thethree-dimensional scene; displaying, by the processor, the image byrendering the image data on a display device; capturing user input atthe processor, wherein the user input comprises motion data representinga motion of a user relative to the display device; modifying, at theprocessor, the image by: computationally isolating a fixation regionwithin the image by computationally processing a subset of image datarepresenting an image object within the image to determine boundaries ofat least one image object associated with the visual fixation point,isolating the at least one image object, and rendering the at least oneimage object on a separate depth layer, wherein the fixation region isdefined by the subset of the image data, and wherein the image object isassociated with the visual fixation point; spatially reconstructing thesubset of image data to computationally expand the fixation region;spatially reconstructing remaining image data relative to the subset ofimage data to computationally compress a peripheral region of the imagerelative to the fixation region in a progressive fashion as a functionof a distance from the fixation region; statically positioning thefixation region relative to a border of the image and moving theperipheral region relative to the fixation region in accordance with themotion of the user to emulate a motion parallax perceived by the humanobserver in the three-dimensional scene; and wherein modification of theimage is modulated by the user input such that a modified image isproduced which synthetically emulates how the human observer wouldperceive the three-dimensional scene.
 2. The method of claim 1, whereinmodifying the image further comprises rotating the image around an axisof the fixation region.
 3. The method of claim 1, wherein modifying theimage further comprises applying a geometrical transformation to theimage.
 4. The method of claim 1, wherein modifying the image furthercomprises altering an angle of view of the image in a horizontal orvertical axis of the image.
 5. The method of claim 4, wherein thefixation region is computationally expanded by an amount positivelycorrelated with the angle of view.
 6. The method of claim 4, wherein theperipheral region is computationally compressed by an amount inverselycorrelated with the angle of view.
 7. The method of claim 1, whereinmodifying the image further comprises computationally isolating thefixation region in response to an indication gesture of the user.
 8. Themethod of claim 1, wherein capturing the user input comprises monitoringa movement of eyes of the user, wherein the movement of the eyes of theuser corresponds to a repositioning of the fixation point within thethree dimensional scene and modifying the image comprisescomputationally isolating the fixation region comprising a repositionedfixation point.
 9. The method of claim 1, wherein capturing the userinput comprises monitoring touch gestures on a touchscreen of thedisplay device, wherein the touch gestures correspond to a repositioningof the fixation point within the three-dimensional scene and modifyingthe image comprises isolating the fixation region comprising arepositioned fixation point.
 10. The method of claim 1, wherein theimage data comprises data relating to a distance between objects and theviewing point in the three-dimensional scene.
 11. The method of claim 1,further comprising updating the display of the display device subsequentto each step of modification of the image.
 12. The method of claim 1,wherein the image data is generated by an optical device comprising alens selected from a group consisting of a rectilinear camera lens, afisheye camera lens, a 360° lens, multiple lenses, and a mechanicallyadjustable lens.
 13. The method of claim 1, further comprisingcomputationally processing, at the processor, the image data so as toapply an image blur effect which progressively increases radially awayfrom the fixation point in at least one of height, width and depth axesof the image.
 14. The method of claim 1, further comprisingcomputationally processing, at the processor, the image data to overlaythe image with an object seen in proximity to a face of the humanobserver in the three dimensional scene.
 15. The method of claim 1,wherein capturing the user input comprises computing, at the processor,a distance between the image and a head of the user.
 16. The method ofclaim 15, wherein the fixation region is computationally expanded by anamount inversely correlated with the distance between the image and thehead of the user.
 17. The method of claim 15, wherein the peripheralregion is computationally compressed by an amount positively correlatedwith the distance between the image and the head of the user.
 18. Acomputer system comprising: user input capturing means configured tocapture user input; a display device configured to display processedimage data; one or more processors; and memory storing thereoninstructions that as a result of being executed by the one or moreprocessors cause the computer system to: provide image datarepresentative of at least a portion of a three-dimensional scene, thethree-dimensional scene being visible to a human observer from a viewingpoint when fixating on a visual fixation point within thethree-dimensional scene; render the image data on to the display deviceto display an image; capture, via the user input capturing means, theuser input, wherein the user input comprises motion data representing amotion of a user relative to the display device; and modify the imageby: computationally isolating a fixation region within the image bycomputationally processing a subset of image data representing an imageobject within the image to determine boundaries of at least one imageobject associated with the visual fixation point, isolating the at leastone image object, and rendering the at least one image object on aseparate depth layer, wherein the fixation region is defined by thesubset of the image data, and wherein the image object is associatedwith the visual fixation point; spatially reconstructing the subset ofimage data to computationally expand the fixation region; spatiallyreconstructing remaining image data relative to the subset of image datato computationally compress a peripheral region of the image relative tothe fixation region in a progressive fashion as a function of a distancefrom the fixation region; statically positioning the fixation regionrelative to a border of the image; and moving the peripheral regionrelative to the fixation region in accordance with the motion of theuser to emulate a motion parallax perceived by the human observer in thethree-dimensional scene; and, wherein modification of the image ismodulated by the user input such that a modified image is produced thatsynthetically emulates how the human observer would perceive thethree-dimensional scene.
 19. The system of claim 18, further comprisingan image capturing means configured to capture the image data whichrepresents the three-dimensional scene.